Smoothed AND Flat Normals for a Mesh

I have an outliner shader which scales your model outward by its normals and renders it a flat color behind the usual diffuse pass, and thus assumes your normals are smoothed. For box-like objects that need flat shading, however, you'd need two sets of normals - a smoothed set for the outliner's pass, and a flattened set for the diffuse lighting.

Games like Red Alert 3 have models that are shaded flat and also outlined, so it's clearly possible. Can one store more than one normal per vertex in Unity? The Genghis Khan way would be to simply import 2 copies of every model in your game, but that'd be a shameful waste of memory to duplicate the vertex positions, uv's and what-not when only the normals differ.

Assuming you let Unity calculate the model's inherent normals flat (since it'd have more vertices that are split) and then calculated a second smoothed set procedurally, how could you pass that second set to the shader for it to scale with?

 

It's not clearly possible based on that, because there's no need for normals for lighting on that model, as there is no lighting. Do you have any examples of "Games like" that use lighting?

 

Really? If you take a look on YouTube (0:38 and onward), as the Mobile Construction Vehicle and other units turn around, you can see that the right sides of them remain in shadow (aside from the dynamic shadows on the ground), indicating a directional light coming from the left. Is that not Lambert diffuse?

 

I'd store the smoothed vertex normals in the vertex colour channel...

 

A viable idea, and just what I tried to do ten months ago when I first attempted this (yes, I've been stuck at this problem for that long). In implementation, however, I'm missing something. Here was my attempt.

First the C# in scripting, which assigns my own calculated normals into the "normals" property of the mesh:

Code (csharp):

1.  
2. Mesh m;
3. //set m's other stuff
4. Vector3[] newNormals=calculate_my_normals();
5. m.normals=newNormals;
6.  

Then, the vertex shader of my outline pass:

Code (csharp):

1.  
2. struct appdata{
3.     float4 vertex:POSITION;
4.     float3 normal:NORMAL;
5. };
6. v2f vertie(appdata v){//supposed to move along normal
7.     v2f o;
8.     o.pos=mul(UNITY_MATRIX_MVP,v.vertex);
9.     float3 norm=mul((float3x3)UNITY_MATRIX_IT_MV,v.normal);
10.     float2 offset=TransformViewToProjection(norm.xy);
11.     o.pos.xy+=offset*_OutlineWidth;
12.     return o;
13. }
14.  

Which produces:



Now, attempting to put the exact same normals into the color channel, I change the scripting to place the x,y,z components of the normals into the r,g,b elements of an array of colors, and assign it to the "colors" property:

Code (csharp):

1.  
2. Mesh m;
3. //set m's other stuff
4. Vector3[] newNormals=calculate_my_normals();
5. Color[] colors=new Color[numVerts];//added
6. for(int i=0;i<verts.Length;i++)colors[i]=new Color(newNormals[i].x,newNormals[i].y,newNormals[i].z,0);//added
7. m.colors=colors;//added
8.  

And change just the one line in the shader:

Code (csharp):

1.  
2. struct appdata{
3.     float4 vertex:POSITION;
4.     float3 normal:NORMAL;
5.     float4 color:COLOR;//added
6. };
7. v2f vertie(appdata v){//supposed to move along normal
8.     v2f o;
9.     o.pos=mul(UNITY_MATRIX_MVP,v.vertex);
10.     float3 norm=mul((float3x3)UNITY_MATRIX_IT_MV,v.color.rgb);//this was previously v.normal
11.     float2 offset=TransformViewToProjection(norm.xy);
12.     o.pos.xy+=offset*_OutlineWidth;
13.     return o;
14. }
15.  

But this results in:


It appears that all the colors are a constant 1,1,1, so they're not getting assigned to properly. What else need I do to assign normals to the color channel?

Thanks.

 

Ah, I think that the vertex colours are clamped 0-1 and stored as 8-bit (as they're supposed to just hold colours, not vector information). So you might find that's not such a viable method.

The Red Alert image you're referencing is using an image effect to create that outline, it's not part of some outline shader on the model.

You could store the x/y in the UV2 and then reconstruct it in the shader?

i.e.

Code (csharp):

1.  
2. Mesh m;
3. //set m's other stuff
4. Vector3[] newNormals = calculate_my_normals();
5. Vector2[] uv = new Vector2[numVerts];//added
6. int i = 0;
7. int limit = verts.Length;
8. for(i = 0; i < limit; i++ ) uv[i]=new Vector2(newNormals[i].x,newNormals[i].y);//added
9. m.uv2=uv;//added
10.  

Code (csharp):

1.  
2. struct appdata{
3.     float4 vertex:POSITION;
4.     float3 normal:NORMAL;
5.     float2 smoothNorm:TEXCOORD2;//added
6. };
7.  
8.  
9. v2f vertie(appdata v){//supposed to move along normal
10.     v2f o;
11.     o.pos = mul ( UNITY_MATRIX_MVP,v.vertex );
12.     float3 norm = float3 ( v.smoothNorm.x, v.smoothNorm.y, 0.0 );
13.     norm.z = sqrt ( 1.0 - norm.x * norm.x - norm.y * norm.y );
14.     norm = mul( (float3x3)UNITY_MATRIX_IT_MV, norm );
15.     float2 offset = TransformViewToProjection ( norm.xy );
16.     o.pos.xy += offset * _OutlineWidth;
17.     return o;
18. }
19.  

 

Thanks Farfarer. With your changes, I now get these errors:

It's been so long since I wrote this I hardly remember if there's somewhere else you need to bind the input channels. Also, I'm not sure why it says "(compiling for flash)". I'm building for web and desktop.

Here's the whole shader, with your changes above:

Code (csharp):

1.  
2. Shader "Custom/Outlined Diffuse"{
3.     Properties{
4.         _Color("Main Color",Color)=(1,1,1,1)
5.         _MainTex("Base (RGB)",2D)="white"{}
6.         _OutlineColor("Outline Color",Color)=(0,0,0,1)
7.         _OutlineWidth("Outline width",Range(0,1))=1
8.     }
9.     SubShader {
10.         Tags{"Queue"="Geometry+202"}
11.         Pass{
12.             Name "OUTLINE"
13.             Tags{"LightMode"="Always" "IgnoreProjector"="True"}
14.             Cull Front
15.             ZTest Off
16.             ZWrite Off
17.             Lighting Off
18.             ColorMask RGB
19.             Blend SrcAlpha OneMinusSrcAlpha
20.             CGPROGRAM
21.             #pragma vertex vertie
22.             #pragma fragment frag
23.             #include "UnityCG.cginc"
24.             struct appdata{
25.                 float4 vertex:POSITION;
26.                 float3 normal:NORMAL;
27.                 float3 smoothedNorm:TEXCOORD2;
28.             };
29.             struct v2f{
30.                 float4 pos:POSITION;
31.             };
32.             uniform float _OutlineWidth;
33.             v2f vertie(appdata v){//moves along normal
34.                 v2f o;
35.                 o.pos=mul(UNITY_MATRIX_MVP,v.vertex);
36.                 float3 norm=float3(v.smoothedNorm.x,v.smoothedNorm.y,0.0);
37.                 norm.z=sqrt(1.0-norm.x*norm.x-norm.y*norm.y);
38.                 norm=mul((float3x3)UNITY_MATRIX_IT_MV,norm);
39.                 float2 offset=TransformViewToProjection(norm.xy);
40.                 o.pos.xy+=offset*_OutlineWidth;
41.                 return o;
42.             }
43.             uniform float4 _OutlineColor;
44.             half4 frag(v2f i):COLOR{return _OutlineColor;}
45.             ENDCG
46.         }
47.         Pass{
48.             Material{
49.                 Diffuse[_Color]
50.                 Ambient[_Color]
51.             }
52.             Blend SrcAlpha OneMinusSrcAlpha
53.             Lighting On
54.             SetTexture[_MainTex]{
55.                 Combine texture*primary DOUBLE
56.             }
57.         }
58.     }
59.     Fallback "VertexLit"
60. }
61.  

Google has slim pickin's. Know what's up?

 

Oh, oops. You might want to change these lines (first one in appdata, second one in vertex shader);

float3 smoothedNorm:TEXCOORD2;
...
float3 norm=float3(v.smoothedNorm.x,v.smoothedNorm.y,0.0);

with these lines;

float4 texcoord1 : TEXCOORD1;
...
float3 norm = float3 ( v.texcoord1.x, v.texcoord1.y, 0.0 );

 

Good, it now compiles, though with the following result:



The side that's off is -Z.

What's your logic with the:"norm.z=sqrt(1.0-norm.x*norm.x-norm.y*norm.y);" line? Something Pythagorean-related, to make it the distance of the hypotenuse between x and y perhaps?

 

The logic is that if you know two of the components of a unit length (normalized) vector, you can use that equation to work out the third component and reconstruct the entire vector.

But it only gets you back a vector that's positive in the axis that you reconstruct, which I didn't think about. Meaning you'd have to store another value to tell you if you need to flip the z or not.

Give this a bash?

Code (csharp):

1. Mesh m;
2. Vector3[] newNormals = calculate_my_normals();
3. Vector2[] uv = new Vector2[numVerts];
4. Color32[] col = new Color32[numVerts];
5. int i = 0;
6. int limit = verts.Length;
7. for ( i = 0; i < limit; i++ ) {
8.     uv[i] = new Vector2 ( newNormals[i].x,newNormals[i].y );
9.     col[i] = new Color32 ( Mathf.Clamp01(Mathf.Sign(newNormals[i].z)), 0.0, 0.0, 0.0 );
10. }
11. m.uv2 = uv;
12. m.colors32 = col;

Code (csharp):

1. struct appdata{
2.     float4 vertex:POSITION;
3.     float3 normal:NORMAL;
4.     float2 texcoord1:TEXCOORD1;
5.     fixed4 color:COLOR;
6. };
7.  
8. v2f vertie(appdata v){//supposed to move along normal
9.     v2f o;
10.     o.pos = mul ( UNITY_MATRIX_MVP,v.vertex );
11.     fixed sign = v.color.r * 2 - 1;
12.     float3 norm = float3 ( v.texcoord1.x, v.texcoord1.y, 0.0 );
13.     norm.z = sqrt ( 1.0 - norm.x * norm.x - norm.y * norm.y ) * sign;
14.     norm = mul( (float3x3)UNITY_MATRIX_IT_MV, norm );
15.     float2 offset = TransformViewToProjection ( norm.xy );
16.     o.pos.xy += offset * _OutlineWidth;
17.     return o;
18. }

 

Rock on. It didn't work with Color32 (typecasting the Mathf.Clamp01(...) with "(byte)" and changing the "0.0" entries to "0") as the "fixed4 color:COLOR" channel didn't seem to be receiving it, but when I change the former back to just "Color", it now works. Thanks a lot.

Know what this image effect is called, or have you perhaps a link to an example?